In today's information intensive environment, there are many businesses and other institutions that need to store huge amounts of digital data. These include entities such as large corporations that store internal company information to be shared by thousands of networked employees; online merchants that store information on millions of products; and libraries and educational institutions with extensive literature collections. A more recent need for the use of large-scale data storage systems is in the broadcast television programming market. Such businesses are undergoing a transition, from the older analog techniques for creating, editing and transmitting television programs, to an all-digital approach. Not only is the content (such as a commercial) itself stored in the form of a digital video file, but editing and sequencing of programs and commercials, in preparation for transmission, are also digitally processed using powerful computer systems. Other types of digital content that can be stored in a data storage system include seismic data for earthquake prediction, and satellite imaging data for mapping.
To help reduce the overall cost of the storage system, a distributed architecture is used. Hundreds of smaller, relatively low cost, high volume manufactured disk drives (currently each disk drive unit has a capacity of one hundred or more Gbytes) may be networked together, to reach the much larger total storage capacity. However, this distribution of storage capacity also increases the chances of a failure occurring in the system that will prevent a successful access. Such failures can happen in a variety of different places, including not just in the system hardware (e.g., a cable, a connector, a fan, a power supply, or a disk drive unit), but also in software such as a bug in a particular client application program. Storage systems have implemented redundancy in the form of a redundant array of inexpensive disks (RAID), so as to service a given access (e.g., make the requested data available), despite a disk failure that would have otherwise thwarted that access. The systems also allow for rebuilding the content of a failed disk drive, into a replacement drive.
Applications operating on a typical storage system may consume networking and processing bandwidth. For example, to detect new material appearing on a typical storage system, a scanning tool needs to continuously scan the storage system for the new material. This scanning process consumes network and processing bandwidth. It further introduces latency from the time the new material is stored on the storage system to the time the scanning tool detects the new material through its scanning process.